Repository navigation
[Core][DBO] Allow dual batch overlap with pipeline parallelism - #68064
Draft
ChengYM5685 wants to merge 24 commits into
Draft
ChengYM5685 wants to merge 24 commits into
ChengYM5685 wants to merge 24 commits into
Conversation
The memory profile does not microbatch, so it sizes only microbatch 0's workspace slots. A later microbatch can need as much scratch as the whole batch (the expert all-to-all receives every rank's tokens), so its slot grew mid-forward: an empty_cache device sync inside a microbatch thread, and once the workspace is locked after graph capture, an AssertionError that kills the thread and leaves its peer parked, hanging the step. After the profile run, grow each lane's microbatch slots to the largest slot of that lane, before capture locks them. The V1 runner's profile run did microbatch, which sized microbatch 0 for half the tokens; an unsplit step then outgrew the locked workspace. Profile unsplit there too, then run split once so the later microbatches also create their persistent resources (e.g. the DeepSeek-V4 ROCm indexer's pooled K buffer) before the lock. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
…efore capture A microbatch thread's hipBLASLt state is per thread and stream, and creating it inside graph capture fails with HIP error 900 (operation not permitted when stream is capturing). The microbatch threads already create it on the compute and comm streams before capture, but models also fork GEMMs onto side streams in capture: DeepSeek-V4 on ROCm runs its compressors on aux streams (execute_in_parallel) and small-batch shared experts on the global aux stream. Capturing a FULL microbatch graph for decode then fails on the first such GEMM. Also run the warm-up GEMM on every CUDA stream the model's modules hold. The streams are found once, by scanning the modules' attributes. The V1 runner's UBatchWrapper only called current_blas_handle(), which does not create the hipBLASLt handle at all; do the same warm-up there. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
…empty Under DP, every rank pads to the largest rank's token count before splitting. A rank whose tokens all fit in the earlier microbatches still splits and runs its all-padding last microbatch, so the expert all-to-all stays collective. Such a microbatch cannot replay a FULL graph captured with real tokens in every microbatch, so the step falls back to eager. The check, `2 * min_tokens < graph_tokens`, missed the rank whose tokens exactly fill the earlier microbatches and only covered two microbatches; use is_last_ubatch_empty instead. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Under DBO, the V1 runner's capture_model captures uniform-decode FULL graphs at or above the decode threshold microbatched only, but profile_cudagraph_memory sampled every graph unsplit. The microbatched graphs cost more: on DeepSeek-V4 (DP8/EP8, MI355X, default thresholds) the graphs took 9.1 GiB against a 4.53 GiB estimate, the KV cache took the difference, and decode at --gpu-memory-utilization 0.92 ran out of device memory. Profile those graphs microbatched, as a group of their own; the condition is shared with _capture_cudagraphs. Each also allocates buffers for its microbatch sizes that outlive the graph pool, so their cost is not shared with the other groups' pool, and it varies too much by size to extrapolate from two samples (from 2 GiB for the largest to 12 MiB for the small ones here; extrapolating overestimated by 22 GiB). Capture every one of them and count their cost in full: 9.0 GiB here, for about 50 s more startup. Clear the UBatchWrapper's graphs afterwards, as the other wrappers' are, so capture_model does not replay a profiling graph. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
A DBO microbatch that lies in the DP padding holds the last request with no query tokens, which the metadata counts as a decode with zero decode tokens. Two ops broke on it: - The sparse decode reshaped an empty index tensor and raised, killing the microbatch thread and hanging the step. Run the decode only when it has rows. - The MXFP4 indexer compressor launched its pooling kernel over the state slot mapping, which spans the DP-padded batch, while the K cache slot mapping and token_to_req_indices it reads per token cover only the real tokens. On this microbatch both are empty and the read faults the GPU (memory access fault at address nil); on other steps the padding tokens read stale entries. Launch over the real tokens only. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Give each DBO microbatch its own MORI EPv2 op: both microbatches have a dispatch or combine in flight at once, and an op's receive and staging buffers are only valid until its next call. prepare() and finalize() run dispatch and combine on the comm stream after yielding to the peer microbatch, so supports_dbo() is true while supports_async() stays false. Graph capture keeps them on the capture stream. The naive DP/EP path does the same; move its _comm_region to ubatching.dbo_comm_region and use it for both. The receive-row bound needs no change: each microbatch's forward context carries its own DP token counts, equal on every rank. Allow --enable-dbo with the mori all2all backend on ROCm when it resolves to EPv2. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Under DBO each microbatch dispatches through its own EPv2 op, and every op was sized for a full batch (max_num_batched_tokens tokens per rank), so the second op's arena doubled MORI's footprint and came out of the KV cache. Microbatch 0's op also serves the steps DBO does not split and keeps the full size. A split step pads every DP rank to the same N <= max_num_batched_tokens tokens and cuts it at N // num_ubatches (or on a request boundary for a uniform spec-decode batch), so a later microbatch holds fewer than N / num_ubatches + (num_ubatches - 1) * q tokens, q being the query length (1 + speculative tokens). Size the later ops to that bound; MORI rejects a batch above an op's size, so an underestimate would fail loudly. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
On GPUs where MORI_ENABLE_SDMA=1 used to select EPv1 (all but gfx1250), a MoRI with EPv2 AsyncLL (ROCm/mori#768) now keeps EPv2 and builds its ops with the AsyncLL kernels over SDMA; an older MoRI still falls back to EPv1. The resolved choice is ParallelConfig.mori_epv2_async_ll. AsyncLL splits dispatch and combine into a send and a receive, so the prepare/finalize supports async: the receive is the hook, which DBO runs on the peer microbatch after its compute is queued (as DeepEP low-latency does), and the shared experts overlap the combine. Each DBO microbatch keeps its own op. The arena is sized from MoRI's AsyncLL region table, and the manager barriers before closing its probe device comm, since ranks trade SDMA queue handles while creating it. With a MoRI that has async_ll_own_sdma_queue (ROCm/mori#892), each op asks for its own SDMA queue: DBO issues both microbatches on the compute stream, where on the shared queue each receive also waits for the other op's send. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Resolve the modify/delete conflicts with vllm-project#63373 (Remove Model Runner V1) by taking main's deletion of vllm/v1/worker/gpu_model_runner.py and vllm/v1/worker/gpu_ubatch_wrapper.py. Their V1-only DBO changes (microbatched CUDA graphs in the V1 memory profile, side-stream hipBLASLt workspaces in the V1 ubatch wrapper, V1 workspace sizing) go away with V1; the V2 counterparts in gpu/model_runner.py, gpu/ubatch_utils.py, ubatch_utils.py and workspace.py are kept.
On main MORI_ENABLE_SDMA=1 always selects EPv1, which is how Quark MXFP4 (OCP MX) MoE models run with --all2all-backend mori. Selecting EPv2 AsyncLL for it by default rejected those models at startup until vllm-project#66733 and vllm-project#66734 land. Keep EPv1 for MORI_ENABLE_SDMA=1 by default and run EPv2's AsyncLL kernels only with an explicit VLLM_MORI_EP_V2=1. Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Each pipeline stage microbatches its own forward pass: non-first stages slice the received intermediate tensors per microbatch and non-last stages merge the microbatch outputs back into one batch before sending, so stages keep exchanging whole batches. The V2 runner already does this (slice_model_inputs / merge_ubatch_outputs); drop the config check that rejected the combination, update the DBO design doc, and test the intermediate-tensor round trip. Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Contributor
|
Documentation preview: https://vllm--68064.org.readthedocs.build/en/68064/ |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Allow dual batch overlap (DBO) with pipeline parallelism in Model Runner V2. Each pipeline stage microbatches its own forward pass and the stages keep exchanging whole batches.
Stacked on #65128 (MORI EPv2 with DBO, native and AsyncLL). Only the last commit is new here; this PR stays a draft until #65128 lands.
Related: #64350 (tracked in #64334) is a draft that allows PP + DBO for one narrower combination (NIXL EP, TP=1, synchronous scheduling, no CUDA graphs) together with a refactor of the microbatch runner, and has only CPU validation so far. This PR is the minimal change for the V2 runner as it is on main: it only drops the config check, covers MORI EPv2 (native and AsyncLL), and brings GPU accuracy and throughput data. If the reviewers prefer to land the scope through #64350, the data below can be used there instead, and I am happy to close this one.
Claims
--enable-dbocan be combined with--pipeline-parallel-size > 1on the V2 runner. The only change to runtime code is removing the config check that rejected the combination; the V2 runner already slicesintermediate_tensorsper microbatch (slice_model_inputs) and mergesIntermediateTensorsoutputs back into one batch (merge_ubatch_outputs).Validation
Hardware and stack: 8x MI355X (ROCm), MORI EPv2 (
VLLM_MORI_EP_V2=1; AsyncLL addsMORI_ENABLE_SDMA=1), AITER MoE, eager (--enforce-eager), modelDeepSeek-V4-Flash(FP8 + FP4 experts). The build under test was this branch plus two MORI EPv2 changes that PP2 x DEP4 needs on this stack and that are under review separately:Server (PP2 x DP4 shown; DEP8 uses
--tensor-parallel-size 1 --data-parallel-size 8):A temporary counter (not part of this PR) in
UBatchRunner.runconfirmed that both stages ran microbatched forwards, and that the second stage's microbatches carried the slicedintermediate_tensors.Accuracy:
lm_eval --tasks gsm8k --num_fewshot 5, all 1319 questions,temperature=0, concurrency 64. The DBO thresholds were lowered to 8 (decode) and 16 (prefill) so that most steps were microbatched.No GPU faults or failed requests in any of these runs or in the mixed-length stress runs that followed each gsm8k.
Prefill throughput:
vllm bench serve --dataset-name random --random-input-len 8000 --random-output-len 1 --ignore-eos --max-concurrency 64 --num-prompts 256 --num-warmups 16 --seed 42, DBO on prefill only (--dbo-decode-token-threshold 100000 --dbo-prefill-token-threshold 512). The four cases were run interleaved in ABCD then DCBA order, with a fresh server per case. The two rounds differ by at most 2.3%. Total token throughput, tok/s:With AsyncLL, mean TTFT for PP2 x DEP4 drops from 3265 / 3359 ms to 2709 / 2725 ms.
CUDA graphs (smoke test only): PP2 x DEP4 AsyncLL + DBO without
--enforce-eager. DeepSeek-V4 falls back toFULL_DECODE_ONLY, and both stages capture. gsm8k scored 0.956 with no failed requests in the stress runs. I did not separately check that decode microbatch graphs were replayed.Unit test:
pytest tests/v1/worker/test_gpu_ubatch_slicing.pygives 30 passed. The new test cutsIntermediateTensorsinto per-microbatch views and merges the microbatch outputs back into the full batch.Details
hidden_statesandresidual) before sending, which is one extra copy per step.tests/v1/distributed/test_pp_dp_v2.pyruns without EP, and the configuration needs MORI on 8 GPUs. Cross-node PP + DBO and decode performance under PP + DBO were not measured.Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
AI assistance (Claude) was used to analyze the code, make the change, and run the validation above. I reviewed the change and the results.